Pattern Kits

Compressed full-text indexes have been one of pattern matching’s most important success stories of the past decade. We can now store a text in nearly the information-theoretic minimum of space, such that we can still quickly count and locate occurrences of any given pattern. However, some files or collections of files are so huge that, even compressed, they do not all fit in one machine’s internal memory. One solution is to break the file or collection into pieces and create a distributed index spread across many machines (e.g., a cluster, grid or cloud). Suppose we want to search such an index for many patterns. Since each pattern is to be sought on each machine, it is worth spending a reasonable amount of time to preprocess the patterns if that leads to faster searches. In this paper we show that if the concatenation of the patterns can be compressed well with LZ77, then we can take advantage of their similarities to speed up searches in BWT-based indexes. More specifically, if we are searching for t patterns of total length m in a distributed index for a text of total length n, then we spend O(m + (g + t) logm) time preprocessing the patterns on one machine and then O ( (g + t) log m log n ) time searching for them on each machine, where g is the size of the smallest straight-line program for the concatenation of the patterns. Thus, if the concatenation of the patterns has a small straight-line program — plausible if the patterns are similar — and the number of machines is large, we achieve a theoretically significant speed-up. The techniques we use seem likely to be of independent interest and we show how they can be applied to pattern matching with wildcards and parallel pattern matching.